Computational models of social and emotional turn-taking for embodied conversational agents: a review

The emotional involvement of participants in a conversation not only shows in the words they speak and in the way they speak and gesture but also in their turn-taking behavior. This paper reviews research into computational models of embodied conversational agents. We focus on models for turn-taking management and (social) emotions. We are particularly interested in how in these models emotions of the agent itself and those of the others influence the agent’s turn-taking behavior and vice versa how turn-taking behavior of the partner is perceived by the agent itself. The system of turn-taking rules presented by Sacks, Schegloff and Jefferson (1974) is often a starting point for computational turn-taking models of conversational agents. But emotions have their own rules besides the “one-at-a-time” paradigm of the SSJ system. It turns out that almost without exception computational models of turn-taking behavior that allow “continuous interaction” and “natural turntaking” do not model the underlying psychological, affective, attentional and cognitive processes. They are restricted to rules in terms of a number of superficially observable cues. On the other hand computational models for virtual humans that are based on a functional theory of social emotion do not contain explicit rules on how social emotions affect turn-taking behavior or how the emotional state of the agent is affected by turn-taking behavior of its interlocutors. We conclude with some preliminary ideas on what an architecture for emotional turn-taking should look like and we discuss the challenges in building believable emotional turn-taking agents. A COMMIT deliverable May 23, 2012